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Summary 


For  many  years,  the  Center  for  Naval  Analyses  (CNA)  has  been 
charged  with  reconstructing  and  analyzing  the  Navy’s  major  fleet 
exercises.  Before  deploying,  a  carrier  battle  group  completes  its  work¬ 
up  cycle  by  participating  in  a  major  exercise  war,  conducted  at  sea 
with  real  ships,  submarines,  and  planes  in  a  scenario  relevant  to  the 
part  of  the  world  to  which  it  will  deploy.  CNA  provides  reconstruction 
and  analysis  support  at  the  request  of  the  fleet  commanders,  who 
need  an  objective  evaluation  of  their  battle  group’s  tactics  and 
readiness. 

On  26  and  27  September  1995,  Commander,  Naval  Base  San  Diego 
(COMNAVBASE)  and  the  Fleet  Industrial  Supply  Center,  San  Diego 
(FISC)  hosted  the  1995  San  Diego  PREP  Area  Exercise.  The  Navy  had 
participated  in  previous  PREP  exercises,  but  this  is  the  first  time  it  had 
led  one.  Because  this  exercise  was  important,  both  in  terms  of  public 
visibility  and  in  developing  a  standard  procedure  for  evaluating 
future  Navy-led  PREP  exercises,  COMNAVBASE  asked  CNA  to  serve 
as  evaluation  director  for  the  exercise,  and  to  provide  the  type  of  ana¬ 
lytical  support  we  routinely  provide  for  major  fleet  exercises. 

This  report  documents  our  evaluation  methodology,  and  offers  a  sug¬ 
gested  evaluation  plan  for  use  in  future  PREP  area  exercises. 


What  is  PREP? 


Section  31 1  of  the  Federal  Water  Pollution  Control  Act  (the  Clean 
Water  Act)  of  1972  required  the  formation  of  a  National  Oil  and  Haz¬ 
ardous  Substances  Pollution  Contingency  Plan  (NCP).  This  resulted 
in  40  CFR-300,  which  set  out  the  National  Response  System  for  deal¬ 
ing  with  oil  and  hazardous  substance  emergencies. 

The  Exxon  Valdez  disaster  of  March  1989  revealed  serious  shortcom¬ 
ings  in  this  nation’s  ability  to  deal  with  a  major  oil  spill,  and  resulted 
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in  the  passage  of  the  Oil  Pollution  Act  of  1990  (OPA-90),  which  man¬ 
dated  revision  of  the  NCR  One  revision  was  to  separate  response 
requirements  for  oil  spills  from  those  for  other  hazardous  substances. 
In  general,  OPA-90  called  for  increased  preparedness  for  major  oil 
spills  by  requiring  formation  of  area  committees  and  area  contin¬ 
gency  plans,  and  periodic  exercises.  It  was  in  response  to  this  require¬ 
ment  for  periodic  exercises  that  the  U.S.  Coast  Guard  (USCG), 
Environmental  Protection  Agency  (ERA),  Research  and  Special  Pro¬ 
grams  Office  of  Pipeline  Safety  (Department  of  Transportation),  and 
the  Minerals  Management  Service  developed  the  Preparedness  for 
Response  Exercise  Program  (PREP). 

For  the  purpose  of  oil  spill  response,  the  country  is  divided  into  60 
areas,  each  of  which  has  its  own  detailed  response  plan,  known  as  an 
Area  Contingency  Plan  (ACP).  The  PREP  program  conducts  20  area 
exercises  per  year,  so  that  each  ACP  is  exercised  once  every  three 
years.  Leadership  responsibility  (organizing,  hosting,  and  assuming  a 
large  share  of  the  funding)  for  each  exercise  is  assigned  by  the  USCG; 
plans  call  for  14  industry-led  and  6  government-led  exercises  per  year. 
Navy-led  exercises  are  considered  industry  led  because,  for  the  pur¬ 
pose  of  oil  spills,  the  Navy  is  simply  a  member  of  the  regulated  com¬ 
munity  and  is  subject  to  regulation  as  is  any  other  industrial 
operation.  Government-led  exercises  are  led  by  the  USCG. 

Each  PREP  exercise  is  put  together  by  a  design  team  composed  of 
representatives  of  each  of  the  m^or  participating  agencies.  PREP 
gfuidelines  [1]  describe  15  “core  components”  of  a  response  which 
should  be  exercised  and  evaluated.  The  design  team  decides  which  of 
the  15  could  be  worked  into  the  exercise,  with  the  local  Coast  Guard 
District  Commander  holding  final  say  in  matters  of  exercise  design. 


Summary 

In  this  report,  we  summarize  the  methodology  we  used  in  serving  as 
evaluation  director  for  the  San  Diego  1995  PREP  area  exercise,  and 
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ofifer  it  as  a  blueprint  for  future  evaluation  efforts.  This  report  sug¬ 
gests  that: 

•  PREP  evaluation  should  focus  on  organizational  issues  and  on 
the  ability  of  the  response  organization  to  execute  the  various 
plans.  The  evaluation  should  examine: 

—  How  the  response  command  structure  was  organized,  and 
how  quickly  and  efficiently  it  was  formed 

—  How  well  it  functioned  once  it  was  formed 

—  The  degree  to  which  the  response  organization  was  able  to 
address  the  tasks  and  priorities  called  for  in  the  contingency 
plans. 

•  An  exercise  such  as  PREP,  in  which  no  oil  is  actually  spilled, 
does  not  permit  an  evaluation  of  how  well  the  spill  contingency 
plan  works.  Those  exercise  objectives  dealing  with  actual 
response  performance,  such  as  discharge  control,  ability  to 
contain  or  recover  oil,  and  ability  to  dispose  of  oil  and  oily 
debris,  cannot  be  directly  evaluated  in  such  an  exercise.  These 
issues  could  be  addressed  by  examining  when  they  were  consid¬ 
ered,  by  whom  in  the  response  organization,  and  what  plans 
were  made  for  dealing  with  these  issues. 

Organization  of  this  report 

This  report  is  organized  as  follows: 

•  The  section  following  this  introduction  comprises  the  bulk  of 
this  report.  In  it  we  discuss  general  evaluation  goals,  and 
present  a  plan  for  addressing  each  of  the  15  standard  evalua¬ 
tion  olyectives  given  in  [1]. 

•  In  the  following  section,  we  describe  the  general  evaluation 
methodology,  and  address  personnel  and  data  requirements. 
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Evaluation  objectives 

Table  1  lists  the  15  core  components  of  a  spill  response  plan,  which 
should  bp  played  and  evaluated  in  a  PREP  area  exercise  [1].  PREP 
guidelines  acknowledge  that  it  might  not  be  possible  to  meet  evefy 
one  of  these  olyeptives  in  a  given  area  exercise.  For  each  exercise,  the 
exercise  design  team  should  decide  which  of  these  components  can 
be  played  and  evaluated  in  its  particular  exercise. 

Table  1 .  PREP  1 5  standard  exercise  objectives  (from  [1  ]) 

Objective 

number  Title/description 

1  Notifications 

2  Mobilization 

3  Operate  within  a  Unified  Command  System 

4  Discharge  control 

5  Assessments 

6  Containment 

7  Recovery 

8  Protection 

9  Disposal 

10  Communications 

1 1  Transportation 

12  Personnel  support 

1 3  Equipment 

1 4  Procurement 

1 5  Documentation 

In  this  section,  we  will  discuss  each  of  these  15  standard  exercise 
objectives,  and  offer  suggestions  for  addressing  them  in  an  evalua¬ 
tion.  For  each,  we  describe  the  issue  and  discuss  ways  to  evaluate  the 
exercise  responses  in  meeting  these  objectives.  In  many  cz^es,  we  use 
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the  San  Diego  Area  Contingency  Plan  as  a  baseline  for  assessing 
events  (i.e„  what  should  happen). 


Notifications 


PREP  guidelines  state  that  the  exercise  should  “test  the  notification 
procedures  identified  in  the  Area  Contingency  Plan  and  the  associ¬ 
ated  responsible  party  plans”  [1]. 

The  initial  notification  process  is  critical  to  an  actual  response. 
Despite  the  obvious  exercise  limitations  in  realistically  playing  out  the 
notification  process,  the  evaluation  should  address  notifications  to 
the  extent  possible. 

Should  there  be  a  real  spill,  the  law  requires  that  the  vessel  or  facility 
experiencing  the  spill  must  call  the  U.S.  Coast  Guard  (USCG) 
National  Response  Center  (NRC);  the  NRC  then  calls  the  local  USCG 
On-Scene  Coordinator  (OSC).  In  the  case  of  San  Diego,  the  USCG 
OSC  is  to  notify  the  State  Office  of  Emergency  Services  (OES).  OES 
then  has  a  long  list  of  agencies  to  notify,  depending  on  the  circum¬ 
stances  of  the  spill. 

For  a  PREP  exercise,  all  players  should  be  required  to  make  the  nec¬ 
essary  phone  calls,  and  the  evaluation  should  assess  the  response  to 
three  questions: 

•  Does  everyone  know  whom  to  call? 

•  How  quickly  are  notifications  completed? 

•  Are  all  the  contacts  listed  in  the  plans  current? 

The  evaluation  should  also  assess  whether  all  necessary  agencies  are 
included  in  the  appropriate  notification  lists.  For  example,  in  San 
Diego,  our  search  of  the  ACP  showed  that  a  few  local  agencies  which 
played  roles  in  the  response  were  not  included  on  notification  lists 
and  thus  were  not  notified  early  in  the  event 


6 


Somewhere  in  the  Incident  Command  System  (ICS),  the  notification 
process  must  be  documented.  The  evaluation  should  address  the 
issue  of  record  keeping  associated  with  the  notification  process. 


Mobilization 


With  respect  to  mpbilization,  PREP  guidelines  state  that  the  exercise 
should  “Demonstrate  the  ability  to  assemble  the  spill  response  orga¬ 
nization  identified  in  the  AGP  and  the  associated  responsible  party 
plans” [1]. 

In  San  Diego,  we  took  a  slightly  more  general  approach  to  this  objec¬ 
tive.  We  interpreted  “mobilization”  to  refer  to  bringing  the  command 
together,  bringing  emergency  assets  in  quickly,  and  performing  those 
tasks  required  within  the  first  two  hours.  In  a  real  spill,  these  initial 
actions  would  be  critical  for  dischaige  control  and  initial  assessment 

For  the  most  part,  mobilization  is  not  really  played  in  exercises  such 
as  PREP.  Because  these  are  not  “no-notice”  exercises,  most  respond¬ 
ers  are  on  scene  inuch  more  quickly  than  they  would  be  in  a  real  spill. 
In  feet  in  San  Diego  players  from  a  few  agencies  were  on  scene  before 
they  were  even  notified. 

We  can  address  rpobilization  in  the  evaluation  by  looking  at  how 
quickly  the  various  cells  in  the  response  oiganization  are  formed  and 
staffed.  If  the  time  required  is  too  artificially  low  because  everyone  is 
already  in  place,  tjie  evaluation  should  look  to  see  if  everyone  knows 
what  must  be  done — according  to  the  plan — within  the  first  two 
hours. 

It  is  also  interesting  to  examine  how  the  plan  is  used  early  in  the 
response.  Do  people  actually  break  it  out  and  look  to  see  what  their 
responsibilities  are?  Does  the  plan  help,  or  get  in  the  way? 

Operate  within  a  UCS 

This  olgective — “Ability  to  operate  within  the  Unified  Command  Sys¬ 
tem”  (UCS) — is  vpry  broad,  and  could  encompass  almost  the  entire 
evaluation. 

What  does  “operating  within  the  UCS”  mean?  It  means  a  UCS  must 
be  formed,  and  all  parts  of  the  UCS  must  accomplish  their  missions. 
This  is  a  bottom-line  objective:  Does  this  structure  facilitate  getting  the  job 
done?  Therefore,  the  evaluation  should  try  to  relate  observations 
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concerning  these  organizational  issues  to  their  impact  on  the  actual 
spill  response. 

Under  this  broad  umbrella,  the  evaluation  could  address  the  follow^ 
ing  issues: 

•  Are  all  individual  agency  plans  and  concerns  combined  into  a 
single  plan? 

•  Do  all  players  work  together  on  a  single  staff,  or  does  each  - 
agency  pursue  its  own  agenda? 

•  How  integrated  are  the  agencies?  Does  information  flow  easily 
among  the  different  agencies  that  make  up  each  cell,  including 
the  UC  cell  and  all  others? 

•  How  well  does  information  flow  among  the  cells?  Is  critical 
information  the  same  when  seen  from  all  cells? 

—  At  the  most  basic  level,  does  the  UC  always  know  what  every¬ 
one  is  doing? 

•  How  does  each  staff  coordinate  and  direct  its  mission?  Can 
each  cell  accomplish  its  assigned  tasks,  or  is  it  overwhelmed? 

Discharge  control 

For  this  objective,  PREP  gfuidelines  state:  “Demonstrate  the  ability  of 
the  spill  response  organization  to  control  and  stop  the  discharge  at 
the  source”  [1]. 

Discharge  control  would  of  course  be  of  great  importance  in  an 
actual  spill,  but  there  isn’t  much  we  can  do  with  this  objective  in  an 
exercise  in  which  there  is  no  actual  spill.  (In  San  Diego  PREP,  the 
vessel  master  handled  all  matters  related  to  discharge  control,  and  no 
one  associated  with  the  exercise  questioned  or  attempted  to  evaluate 
the  actions  of  the  vessel  master  in  handling  his  ship.) 

In  San  Diego  PREP,  we  simply  noted  how  and  when  this  was  consid¬ 
ered  by  the  UC,  and  how  much  effort  was  directed  toward  it  until 
exercise  control  declared  the  discharge  to  be  under  control. 
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Assessments 


This  olgective  refers  both  to  the  initial  assessment  of  the  spill  and  to 
continuous  assessments  of  how  the  response  is  proceeding. 

The  evaluation  should  note  the  time  of  the  first  assessment  of  spill 
size  and  the  dme  at  which  this  initial  estimate  is  updated.  Not  much 
can  be  done  regarding  the  accuracy  of  the  initial  assessment;  because 
there  is  no  actual  spill,  the  assessment  consists  of  the  exercise  control¬ 
lers  telling  the  player  in  quesdon  what  they  “saw  as  they  flew  over  the 
spill  in  a  helicopter.” 

The  evaluadon  could  try  to  determine  how  often  assessments  are 
updated.  In  a  one-day  exercise,  we  won’t  see  many  updates,  but  per¬ 
haps  we  can  determine  what  the  long-term  intendon  is  in  this  regard. 

The  evaluadon  should  examine  who  performs  the  assessments  and  to 
whom  results  are  sent.  Does  informadon  from  the  periodic  assess¬ 
ments  flow  to  all  those  in  the  response  organizadon  who  could  bene¬ 
fit  from  it?  The  most  important  issue  is:  How  are  the  assessments 
acted  upon?  The  evaluation  should  look  for  specific  instances  in 
which  the  assessment  led  to  changes  in  plans  or  operadons. 


Containment 


For  this  olgecdve,  PREP  guidelines  require  the  exercise  to  “Demon¬ 
strate  the  ability  of  the  spill  response  organizadon  to  contain  the  dis¬ 
charge  at  the  source  or  in  various  locadons  for  recovery”  [1]. 

Initial  containment  is  one  of  the  highest  priori des  in  an  oil  spill 
response.  However,  in  an  exercise  there  is  no  oil  to  contain.  To 
address  containment,  the  evaluadon  should  discuss  the  proposed 
booming  strategy,  pardcularly  that  which  occurs  immediately  after 
the  spill.  If  some  boom  is  actually  deployed  at  the  site  of  the  exercise 
spill  (as  was  the  case  in  San  Diego),  the  dmeliness  of  that  evoludon 
should  be  addressed. 
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Recovery  and  disposal 

With  respect  to  recovery,  the  PREP  guidelines  state  that  we  needn’t 
actually  pick  up  any  oil.  We  simply  must; 

•  Demonstrate  the  ability  to  assemble  and  deploy  the  on-water 
resources  identified  in  the  Incident  Action  Plan  (lAP) 

•  Demonstrate  the  ability  to  assemble  and  deploy  the  shoreside 
cleanup  resources  identified  in  the  lAP. 

With  respect  to  disposal,  we  must: 

•  Demonstrate  the  ability  of  the  spill  response  organization  to 
dispose  of  the  recovered  material  and  contaminated  debris. 

No  oil  is  actually  recovered  or  disposed  of  during  a  PREP  exercise,  so 
there  isn’t  much  the  evaluation  team  can  do  with  these  issues. 

Due  to  the  volatility  of  oil  (particularly  light  oils,  such  as  DEM),  and 
its  tendency  to  disperse  throughout  the  water  column,  the  best  time 
to  recover  oil  from  the  water  is  as  soon  as  possible  after  it  is  spilled. 
Therefore,  recovery  success  is  highly  dependent  on  the  ability  to  get 
recovery  equipment  in  action  as  soon  as  possible. 

The  evaluation  should  note  what  the  ACP  called  for  with  respect  to 
recovery  and  disposal.  To  evaluate  recovery,  an  assessment  of  how 
quickly  assets  could  be  brought  on  scene  could  be  attempted.  For  dis¬ 
posal,  the  evalviation  could  comment  on  when  and  how  the  disposal 
task  was  considered  by  the  response  oi^nization. 


Protection 


Protection  is,  of  course,  the  main  thrust  of  the  exercise.  This  broad 
objective  consists  of  several  more  specific  objectives: 

•  Protective  booming 

•  Dispersants,  in-situ  burning,  and  bioremediation 

•  Water  intake  protection 

•  Wildlife  recovery  and  rehabilitation 

•  Population  protection. 
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To  evaluate  booming,  the  evaluadon  should  trace  the  thought  pro¬ 
cess  behind  the  booming  strategy  employed.  For  example,  immediate 
containment  booming  is  not  appropriate  in  cases  where  there  is  a 
potential  for  fire  or  explosion.  The  evaluation  could  look  to  see 
whether  this  or  similar  issues  were  considered.  The  evaluadon  should 
also  compare  the  exercise  booming  strategy  with  the  protecdon  pri- 
orides  set  forth  in  the  AGP  and  the  fecility  response  plan. 

Dispersants,  burning,  and  bioremediadon  are  to  be  used  only  in  cases 
where  the  preferred  method  (pickup)  is  inadequate.  Decisions  on 
the  use  of  these  methods  must  be  reached  early  in  the  response.  The 
evaluadon  should  look  at  when  these  opdons  were  considered,  by 
whom,  when  a  decision  was  made,  and  why. 

Water  intakes  are  among  the  highest  priorides  of  the  “economically 
sensidve”  areas.  For  this  olgecdve,  the  evaluadon  could  report  if  and 
when  this  issue  was  considered.  If  there  was  indeed  a  potendal  threat, 
was  it  accorded  the  priority  dictated  by  the  plan,  and  what  acdons 
were  taken?  Similarly  for  wildlife  protecdon  and  recovery,  the  evalua¬ 
don  should  ask:  When  and  how  were  these  issues  considered? 

Of  course,  population  protecdon  is  the  highest  of  all  priorides.  The 
evaluation  must  look  at  what  decisions  were  made  in  this  regard. 
When  and  how  were  public  health  concerns  accounted  for?  Was  there 
any  additional  information  the  UC  would  have  needed  to  assess  the 
potendal  threat  to  hmnans? 

Communications 

This  objective  includes  both  internal  (within  the  command)  and 
external  (to  units  out  on  the  water)  communications.  Communica¬ 
tions  and  the  related  information  flow  are  critical  to  the  success  of  a 
complex  command  structure  such  as  an  ICS,  and  will  be  a  high  prior¬ 
ity  of  the  evaluation. 

Comparing  the  information  held  in  the  various  cells  is  a  useful  tool 
for  assessing  the  flow  of  information.  An  example  of  one  tool  for 
doing  this  is  shown  in  figure  1,  from  the  San  Diego  evaluation.  It 
shows  the  volume  of  oil  spilled,  as  a  function  of  time,  as  seen  from 
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four  views:  the  true  volume,  based  on  controller  updates  provided  to 
the  vessel  master  at  the  scene;  the  volume  assumed  in  the  U/C  Cell; 
the  volume  assumed  in  the  Operations  Cell;  and  the  volume  assumed 
in  the  Planning  Cell.  In  a  perfect  organization,  this  critical  informa¬ 
tion,  on  which  presumably  almost  all  decisions  would  be  based, 
should  flow  immediately  from  the  vessel  to  the  U/C  Cell,  then  out  to 
all  other  cells. 


Figure  1 .  Volume  spilled,  as  seen  from  four  views 
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The  flow  of  this  information  from  the  vessel  to  the  U/C  went  reason¬ 
ably  well,  but  the  flow  from  the  U/C  to  the  Operations  Cell  did  not 
go  as  well.  During  a  critical  4-hour  period  early  in  the  response.  Oper¬ 
ations  thought  they  were  dealing  with  a  125,000  gallon  spill  when  in 
feet  U/C  knew  the  volume  was  265,000  gallons. 

The  evaluation  should  generally  evaluate  how  effectively  the  commu¬ 
nications  load  was  managed.  Were  sufficient  communication  lines 
(phones)  available  in  the  command  spaces,  or  were  they  constantly 
overloaded?  How  were  internal  communications  passed:  by  phone,  or 


simply  by  word  of  mouth?  If  any  computer-based  information  systems 
were  used,  their  utility  should  be  documented. 


Transportation 


With  respect  to  this  objective,  PREP  guidelines  state  that  we  must: 
“Demonstrate  the  ability  to  provide  effective  multi-mode  transporta¬ 
tion  both  for  execution  of  the  discharge  and  support  functions”  [1] . 

In  a  typical  ICS  organization,  transportation  is  primarily  the  responsi¬ 
bility  of  the  Logistics  section.  They  are  to  ensure  prompt  delivery  of 
necessary  resources,  and  keep  the  UC  informed  on  the  status  of  all 
procurements. 

An  analysis  of  logistics  issues  should  start  by  looking  at  information 
flow:  Who  tells  the  Logistics  Cell  what  assets  to  move,  and  when?  Is  the 
Logistics  Cell  given  severe  time  constraints?  After  arranging  transporta¬ 
tion,  what  does  Logistics  brief  to  the  UC?  Does  the  UC  then  use  this 
information  in  adjusting  plans  and  operations? 

The  use  of  newly  arriving  assets  should  also  be  evaluated.  Is  tasking 
coordinated  well  in  advance,  so  that  when  assets  are  brought  in  they 
are  put  right  to  work,  or  could  we  have  a  situation  in  which  assets  are 
on  hold  while  awaiting  tasking? 

A  documentation  of  the  time  it  takes  for  assets  to  actually  arrive  on 
scene  would  be  very  informative.  However,  it  is  generally  hard  to 
derive  this  information  from  a  PREP  exercise  because  the  involved 
agencies  know  about  the  exercise  ahead  of  time,  and  in  fact  often 
have  equipment  and  people  in  place  before  the  time  of  the  spill. 

Equipment  maintenance  and  support 

This  olyective  calls  for  exercise  participants  to  “Demonstrate  the  abil¬ 
ity  to  maintain  and  support  all  equipment  associated  with  the 
response”  [1]. 

A  PREP  one-day  area  exercise  does  not  allow  us  to  do  very  much  with 
this  objective.  For  San  Diego,  we  simply  noted  which  agencies  pro¬ 
vided  response  equipment,  and  whether  or  not  they  brought  support 
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infrastructure  with  them  (in  the  case  of  out-of-town  agencies) .  (In  the 
case  of  the  Navy,  SUPSALV  assets  are  always  accompanied  by  an  exten¬ 
sive  support  infrastructure.)  The  evaluation  could  note  whether  or 
not  this  issue  was  considered  in  the  response  command  organization, 
and  if  so,  how. 

Personnel  support 

The  personnel  support  objective  includes: 

•  Management 

•  Messing  and  berthing 

•  Operational  and  administrative  spaces 

•  Emergency  procedures. 

Evaluation  of  management  issues  could  include  a  discussion  of  the 
time  needed  to  organize  the  command  structure.  Management  also 
includes  coordinating  personnel  assignments,  developing  the  watch 
bill  for  headquarters  staff,  and  moving  people  in  and  out  of  response 
headquarters.  In  the  evaluation,  any  observation  about  administrative 
logistics  (i.e.,  people  getting  lost  and  not  knowing  where  their  spaces 
are,  badge  confusion,  etc.)  should  acknowledge  that  this  is  to  be 
expected  on  opening  day,  and  no  doubt  would  rapidly  shake  out  in 
another  day  or  two.  The  evaluation  should  note  a  problem  along 
these  lines  only  if  it  appears  a  real  issue  exists. 

For  a  spill  in  a  major  city  or  a<^acent  to  a  large  military  installation, 
finding  bunks  and  meals  for  the  responders  should  not  be  a  major 
problem.  A  paragraph  or  two  about  where  and  how  people  would 
actually  be  lodged  and  fed  is  probably  sufficient 

The  evaluation  should  report  on  the  layout  of  response  headquarters 
spaces,  whether  or  not  staff  spaces  were  large  enough,  and  whether 
there  were  any  obvious  problems  with  the  layout.  This  information 
will  be  available  in  player  debriefe  which  occur  immediately  following 
the  exercise.  If  possible,  the  evaluation  should  address  how  well  the 
layout  facilitated  information  flow  among  cells. 
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Emergency  procedures  are  generally  the  responsibility  of  the  safety 
staff.  The  evaluation  shotild  look  at  the  issues  considered  by  the  safety 
staff: 


•  What,  specifically,  will  personnel  be  doing  that  is  dangerous? 

•  What  are  the  potential  injuries? 

•  Is  there  enough  danger  to  warrant  standby  medical  teams? 

•  Are  there  standards  in  this  industry  for  on-scene  medical  sup¬ 
port,  and  if  so,  are  they  met? 

The  evaluation  should  note  when  and  by  whom  these  issues  are 
addressed. 


Procurement 


This  objective  covers  procurement  of  personnel  and  of  resppnse  and 
support  equipment. 

In  San  Diego  PREP,  timely  procurement  of  response  gear  was  often 
impeded  by  communication  problems  between  the  Operations  Cell, 
which  decides  what  response  equipment  is  needed,  and  the  Logistics 
Cell,  which  locates  the  equipment  and  orders  it  An  evaluation  of  pro¬ 
curement  issues  must  include  a  look  at  the  ever  present  issue  of  infor¬ 
mation  flow,  which  is  of  course  crucial  to  every  function  of  the 
response  organization.  Some  of  the  specific  questions  to  look  at  are: 

•  Who  makes  the  decision  to  bring  in  people  and  gear  and  when? 

•  To  whom  is  this  decision  communicated? 

•  How  quickly  is  it  acted  upon? 

•  What  agencies  do  the  appropriate  staff  members  contact  to 
acquire  the  needed  p>eople  or  gear? 

•  Who  keeps  records  of  what  equipment  is  en  route,  and  how 
much  it  will  cost? 


Documentation 

The  evaluation  team  will  keep  a  close  eye  on  documentation  because 
it  is  as  important  to  the  evaluation  process  as  it  is  to  the  play  of  the 
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exercise.  The  quality  of  the  evaluation,  and  to  a  large  extent  the  use¬ 
fulness  of  the  exercise,  is  directly  proportional  to  the  quality  of  the 
dociunentation. 

In  the  standard  ICS  organization,  the  primary  responsibility  for  doc¬ 
umentation  lies  with  the  Information  Management  group,  which  is 
part  of  the  UC  staff.  Of  course,  all  cells  must  document  major  events 
and  decisions.  At  the  basic  level,  documentation  should  allow  the 
evaluation  team  (or  outside  investigators  in  the  case  of  a  real  inci¬ 
dent)  to  reconstruct  everything  that  happened  and  determine  the 
logic  and  information  on  which  all  major  decisions  were  based. 

All  cells  should  keep  watch  logs  and  communications  logs.  The  com¬ 
munications  log  will  record  all  information  passed  or  received.  Paper 
traces,  or  some  similar  type  of  visual  display  of  the  oil  slick  and  all 
assets,  would  be  of  great  use  both  for  documentation  purposes  and  to 
help  the  UC  keep  the  “tactical  picture”  during  exercise  play.  The  util¬ 
ity  of  any  computer-based  information  management  system  used  by 
the  response  organization  should  also  be  assessed. 

Data  and  record  keeping  requirements  must  be  worked  out  between 
the  evaluation  director  and  the  exercise  control  team  before  the  exer¬ 
cise,  to  ensure  that  all  logs  and  records  needed  for  the  evaluation  are 
being  collected  during  exercise  play. 


Evaluation  process 

In  this  section,  we  will  describe  the  overall  evaluation  methodology 
and  discuss  the  personnel  and  data  needed  to  perform  the  evalua¬ 
tion. 

General  methodology 

The  key  features  of  exercise  evaluation  are  obyectivity  and  openness. 
If  some  things  in  the  Area  Contingency  Plan  don’t  work,  or  personnel 
aren’t  able  to  execute  the  plan,  the  evaluation  must  point  this  out 
Better  to  know  now  what  doesn’t  work  than  later,  when  the  cost  could 
be  disastrous. 

Simply  stated,  the  general  evaluation  procedure  is: 

•  Observe  events  and  collect  data 

•  Reconstruct  and  analyze 

•  Document  results  (through  briefings  and  a  report) . 

The  evaluation  will,  of  course,  dociunent  what  happened,  in  as  much 
detail  as  possible.  The  evaluation  team  should  also  try  to  determine  why 
things  happened,  and  why  decisions  were  made.  To  do  this,  it  must  ana¬ 
lyze  information-related  issues  and  the  decision-making  process. 
During  the  exercise,  the  evaluators  should  carefully  note  the  logic  and 
information  on  which  m^or  actions  were  based. 

The  evaluation  should  avoid  subjective  remarks  which  can’t  be 
backed  up  with  examples  or  facts.  The  aim  is  to  evaluate  the  response 
system  and  its  component  tactics — not  the  performance  of 
individuals. 

Our  methodology  for  evaluating  the  San  Diego  PREP  Area  Exercise 
was  basically  that  which  we  regularly  use  in  our  analyses  of  fleet  exer¬ 
cises.  We  placed  members  of  our  evaluation  team  throughout  the 
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response  organization  to  observe  events  and  take  notes.  At  the  end  of 
the  exercise,  we  collected  copies  of  logs  maintained  in  the  various 
cells,  copies  of  status  boards  and  press  briefings,  and  participated  in 
debriefe  of  players.  A  few  members  of  our  evaluation  team  remained 
on  site  for  about  a  week  after  the  exercise  to  reconstruct  events  and 
prepare  a  rough  draft  evaluation  report  The  evaluation  director  con¬ 
solidated  all  inputs  and  then  completed  the  report. 

Reference  [2]  describes  our  San  Diego  evaluation  methodology  in 
more  detail,  and  includes  lists  of  suggested  events  that  the  evaluators 
in  each  cell  should  look  for.  Reference  [3]  is  the  San  Diego  formal 
evaluation  report. 

Evaluator  assignments 

To  observe  events  and  understand  the  response  organization,  mem¬ 
bers  of  the  evaluation  team  must  be  placed  in  the  main  cells  of  the 
response  organization.  Figure  2  shows  suggested  evaluator  coverage. 
In  addition,  evaluators  should  be  placed  at  the  scene  of  the  accident 
(if  there  is  one  in  the  exercise)  to  observe  immediate  response,  and 
at  the  local  Coast  Guard  Marine  Safety  Office,  to  monitor  notifica¬ 
tions  and  immediate  Coast  Guard  actions. 


Figu  re  2 .  Eva  I  uator  coverage 
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Evaluators  should  be  in  position  early,  in  order  to  observe  the  mobi¬ 
lization  process.  During  exercise  play,  they  should  take  extensive 
notes,  recording  the  time  of  all  observations.  These  notes  should 
allow  a  reader  to  understand  what  each  cell  was  doing  and  why  they 
were  doing  it 

To  prepare  for  the  exercise,  all  members  of  the  evaluation  team 
should  read,  at  a  minimum: 

•  The  PREP  guidelines  [1] 

•  The  Area  Contingency  Plan 

•  The  responsible  party’s  facility  plan  (in  the  case  of  a  Navy-led 
exercise,  the  facility  plan  and  the  NOSC  plan). 

Most  importantly,  evaluators  must  remember  to  think  of  the  big  pic¬ 
ture.  As  they  observe  events  during  exercise  play,  they  should  try  to  tie 
events  to  the  thing  that  really  matters:  the  ability  of  the  responding 
group/oiganization  to  effectively  respond  to  the  oil  spill. 


Required  data 


It  is  critical  that  the  exercise  design  team  require  all  cells  to  keep  watch 
logs  and  communications  logs,  documenting  all  actions  and  informa¬ 
tion  passed  and  received.  Because  communications  will  be  over  regular 
telephones,  the  evaluators  in  each  cell  will  not  hear  conversations,  so 
communications  logs  are  especially  important  for  tracking  the  flow  of 
information. 

Any  charts  or  paper  traces  showing  locations  of  the  oil  slick  and 
response  assets  would  be  extremely  helpful. 

A  basic  feet  of  exercise  reconstruction  and  evaluation  is  that  the  qual¬ 
ity  of  the  evaluation  product  is  highly  dependent  on  the  quantity  and 
quality  of  the  information  made  available  to  the  evaluation  team.  In 
summary,  the  following  are  required  for  an  effective  evaluation: 

•  Extensive  evaluator  notes 

•  Copies  of  all  logs  and  briefings,  including: 
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—  Cell  watch  logs 

—  Cell  communication  logs 

—  Press  briefings 
Copies  of  all  status  boards 

All  paper  traces/charts  used  to  track  the  spill. 
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